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FOREWORD 



A central problem facing TAFE in t;^e iSSOs ii how to react quickly to the 
educational needs which arise from technologicai change. In 1983 the tAFE 
National Centre for Research and Development commissioned the authors, 
employed by NSW TAFE, to investigate curriculum research methods which held 
promise of speeding up the research phase of curriculum development. This is 
a summary of the four reports which comprised that study: 



Anderson, T., & Jones, N. (1986). TAFE currlcuJum r esearch- A 
review of groujj process methods. 

Anderson, T., & Jones, N. (1986). TJi e facilitati on of curriculum 
research Workshops in TAFE. 

Jones, N., Anderson, T., Dawson, D., & Dowling, R. (1986). 
curriculum rese arch: A revie w of group pro cess methods. 
Descriptive bibliography. — • 

Jones, N., & Anderson, T. (1986). TAFE curr iculum resea rch: A 
j^evieAv of group process methods. Resjarch^esign. " — 

Copies of the reports can be obtained through the TAFE Clearinghouse, or 
from Nelson Wadsworth, 480 La Trobe street, Melbourne, Victoria, 3001. 
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ABSTRACT 



This paper summarises four reports arising from a study of wiys in which 
technical and further education curriculum research can be speeded up in the 
data collection and analysis phases. The study was commissioned by the TAFE 
National Centre for Research and Development, Adelaide, Australia. This paper 
provides descriptions of research methods reviewed, including exploratory 
research, the Search Conference Method, Delphi, DACUM, Critical Incident 
Technique, and Force Field Analysis; summarises the procedures entailed in 
management of group process curriculum research methods, lists the 
bibliographic sources used, and provides tables of contents for the four 
reports. 
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!• SUMfvlARY OF TAFE CURRIGUttlM RESEARCH METH6C«: A REVIEW 
OF GROUP PROCESS METHOE^* 

LI ABSTRACT 

This TAFE national project examines ways in which technical and further 
education curriculum research can be speeded up in the data collection and 
analysis phases. After discussing this neisd for preliminary investigation the 
study describes and analyses the following research procedures: exploratory 
(qualitative) research, the Search Conference Method, the DACUM (developing a 
curriculum) Method and derivatives of DACUM, the Nominal Group Technique, 
the Delphi Method, the Critical Incident Technique and Force Field Analysis. 
The methods were chosen after reviewing the literature and research methods 
used in TAFE. 

Each method has been presented with the aim of providing intending users 
with sufficient information about the applications of the method, the data 
produced, how the method is used, and what are its pitfalls. Where possible, 
details of duration, cost, time and resources needed, have been included. Since 
these methods provide different perspectives on occupational analysis and 
therefore yield different types of data, some ways in which they can be used 
in combination are covered. A guide to the selection of the methods is 
provided. 

The planning and implernentation of curriculum grdUp process methods is 
summarised in terms of the value orientations (overarching principles which 
guide choices between alternatives) and social rules (practical guides for action) 
which a group facilitator may adopt. A model of the general stages and steps 
in planning and conducting curriculum research workshops is included. Some 
issues which arise in the use of group process methods, such as strategies for 
'conflict resolution*, are highlighted. 

Appendices cover some ques*:ionnaire mechods for occupational analysis, a 
program for a short Search Conference, a brief description of an observational 
method for the study of skill masters, and some examples of DACUM charts. 

The methods reviewed offer, in varying degrees, ways of shortening the 
curriculum research and data analysis process. The Search Conference, Delphi 
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and exploratory methods and, to a lesser extent, the Nominal Group Technique 
(isIGt) hold promise iFor providing data to enable curriculum to be adapted to 
occupational structural change. 

The NGT method appears to offer a useful approach for identifying the 
issues to be addressed in curriculum development and is a useful 'conflict 
resolution' procedure in certain situations. 

The DACUM method, regarded as a quick and effective means for analysing 
occupational education needs at the psycho-molor level, holds promise, when 
linked to Delphi-type processes, of providing data in the cognitive domain (e,g, 
central educational concepts and the knowledge/skills needed to carry these 
concepts into effective task performance on the job)., DAGUM is also 
promising for detailing "the here and now' of work functions but, in its 
traditional form, appears less effective, compared with the Search Conference, 
Delphi and exploratory research, in detecting change within occupations. 
DACUM, therefore, may need to be linked to other methods more sensitive to 
change when used as a curriculum research procedure on occupations 
undergoing structural change. Or it may need to inciude an additional segment 
which looks at trends within the occupation, possibly at the end of the DAGUM 
session. 

The Critical Incident Technique is seen as offering advantages in curriculum 
research for short courses, especially where urgent information is sought on the 
immediate, practical knowledge needed by students. It can also be used in 
assessing what 'critical' know^ledge needs to be included in a course for 
assessing students* perceptions of the relevance of a program to their needs. 

rbrce Field Analysis, since it enables a planning gro^^p to nlghlight :h2 
strengths and weaknesses of the forces affecting the implementation of change, 
appears best suited for use as an adjunct to other group process methods or in 
teacher workshops examining how occupational data can be translated into a 
curriculum. 

1.2 QUESTIONS WHICH GUIDE SEtECTieN OF CURRICULUM RESEARCH 
METHODS 

Table 1 sets out a number of issues and questions which guide selection of 
the methods covered by this study. Since each method has specific strengths 
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and weaknesses a summary of the main features of the methods is given in 
SiBCtibh 1.4 arid 1.5 prior to the detailed descriptions of the methods in 
ehapters 2 and 3, of the report 'TAFE curriculum research: a i^eview of group 
prdcisss methods'. 

Broadly, there are three sets of issues to be considered in selecting a 
research strategy. These concern content, methodology and the problems 
associated with introducing change. Content issues include: 

1. The 'what' versus the 'how' of curriculum. 

2. The relative emphasis which should be placed on identifying and learning 
skills as opjjosed to understanding the underlying concepts. 

3. The relative emphasis which should be placed on identifying the future 
directions in which an industry is moving and emerging skill requirements, as 
opposed to analysing the 'here and now' of work in an industry to identify the 
skills and knowledge currently heeded. 

Methodological issues include: 

1. The relative strengths of qualitative and quantitative research methods. 

2. Sarnpling bias in the selection of key informants and workshop 
participants and the accuracy/reliability of data so obtained. This is equally an 
issue for all methods. 

3. To what extent should educators as well as practitioners be involved in 
group process research methods? 

4. Who controls the workshop output (the tension between researcher 
directed focus and participant directed focus). 

Problems of introducing change are complex but two issues need to be 
highlighted: (A) the problem cf providing a sound rationale for resource 
allocation in times of const.-ai-.cs on funding and (B) the problems of selecting 
a curriculum research and development process which will facilitate 
implementation of a curriculum. The latter issue is related to the so-called 
•change agent' role of researchers and edacators and opens up questions of 
what is an appropriate consultancy role relationship between researchers, 
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curriculum developers and curriculum implismisntisr?. (See Appendix B of the 
main report for an example of a cdrisultahcy model used for a fast-response 
curriculum development project in NSW TAFE.j 

There are no easy resolutions to any of these issues. Goriterit issues are 
taken up again in Chapter 4 of the main report when we discuss ways of 
bleridirig research methods. Methodology issues are taken up in Chapter 6. 
Table 1 lists some key questions to be borne in mind when selecting research 
methods. 



Table 1 



Questions which guide selection of w-called 'fast-fesporise' curriculum research 
methods 



Key Question 



Curriculum Research Method 



What is currently available that 
bears on curriculum development 
in the arisa? 

bo you heed to describe the 
general field to be covered by 
a course and to identify the main 
elements in this field and their 
interrelatibriships, or to 
identify the major questions 
which curriculum research 
should address? 

Is the occupation undergoing 
substantial change? 



Is the_occupation relatively 
stable? 

Is there group tension about 
the direction a course 
review should take? 



Is urgent information needed 
about the critical knov^^edge 
needed to perform specific work 
functions? 

Does the data derived from any 
of the methods require 
validation or does the derived 
program require validation? 



Preliminary 
investigation 

Exploratory research 
Student profiling 
Graduate student survey 



Exploratory research 
The Delphi Method 
The Search Conference 

DACUM (or derivatives) 



Search Conference 
Nominal Group Technique 
(Possibly Force Field 
.Analysis) 



Critical Incident Technique 



Questionnaire 



1.3 SUMMARY OF METHODS 
L3,l Preliminary- inyestAgatiori 

While preliminary research is, by nature exploratory, a distinction will b = 
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made between preliminary work of the kind described here and the techniques 
of exploratory (qualitative) research described in Seetibri U3.2. 

All curriculum development projects in TAFE should include some form of 
preliminary investigatidn. Preliminary investigation is important because 
sufficient information may be obtained which simplifies the curriculum research 
project by showing where gaps exist in available knowledge. Moreover, suitable 
curriculum materials or even a syllabus may be turned up during the 
preliminary investigation phase, enabling research strategies such as pre-testing 
or other forms of validation to be used if confirmation of the relevance of the 
syllabus is needed. 

The scope of preliminary work varies, of course, according to time and 
other constraints and may touch on some or all of the following areas, 
depending on the size and importance of the project. 

(a) Investigation of the relevance to curriculum design and implementation of 
legal or regulatory requirements bearing on the occupation, including awards, 
codes of safe practice. 

(b) Identification of the strengths and weaknesses in the existing educational 
program including: views of teachers; evaluation of outcomes for graduates of 
the program; and development of a profile of students which may include: 



. Age, sex and educational backgrburid (including 
literacy and numeracy levels); 

. Work experience; 

• Work performed by the employer (broad categories); 
. Work performed by the student: 

- specific processes performed 

- variety of work performed 

- extent of cdntrbl, if any, over the nature 
of the work including its scope and depth 

- opportunity for making, decisions about the tasks 
performed^ i.e. scope for putting own ideas 
into practice 



- students* views about shifts/trends within 
the dccUpatibri 

- students* views about their career paths (e^gi 
stay 'on-tbdls*, move into management or 
sUpervisibri^ start own business) 

- students' views abbut strengths and weaknesses 
in the education received, including problems 

being encountered with the order in wh[ch the course 
is traversed, addressing, if needed, questions of 
whether knowledge which students see as vital to be 
acquired ear / in the program is placed, in the[r 
view, too far into the course (see Critical Incident 
Technique) 

- students' reasons for abandoning training 

- examination of students* use of other educational 
souicesi 

(c) Collection of relevant data frbm: Australian Bureau of Statistics (ABS); 
Australian Standard Classification of Occupations (ASCO); Kdmpass (industry) 
directory; research reports, e.g. sUrvey or review reports, workforce forecasts, 
occupational analyses, instruction/training manuals^ curriculum dbcuments, 
teacher reports, theses, recbrds bf advisory committee meetings^ 

(d) Identification of other curriculum work in the same br related fields, 
through literature searches including the TAFE National Clearinghouse 
publications: Initiatives in TAF-E and Projects in Pr ogress. 

(e) Consideratibn bf TAFE policy in relation to the area (e.g. equality of 
opportunity, occupational health and safety, etc). SbUrces of TAFE policy 
including policy ahd/br cUrricUlUm units within TAFE; policy statements by 
senior officers; CTEC Triennium reports (TAFEC); Cbmmittee of Enquiry 
reports from other gbverhment departments. 

(f) Identification and consultation with key infbrmants with a broad grasp or 
detailed khbwledge bf the area. 

(g) Identification and consultation with decisibn-makers, client groups, 
audiences arid stakehblders; 



withiri TAFE, sUch as schools, colleges^ _regions, teachers, 
studerits, senibr bfficers, curriculum specialists; 
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. outside TAFE, such as empioyer and empidyee groups, 
other government departments, other educatidrial 
providers, Natidnal Training Council, Industry 
Training Committees (see sectidn dn expldratory 
research). 

(h) Identification of the key influences which are impinging on the 
occupational area (techndlogicaU ecdndmic, structural, social) especially changes 
which are likely to take effect during the next ten years (see sections on 
exploratory research, the Search Conference and the Delphi methods). 

(i) Clarification of the areas which the research needs to cover and 
formulation of the research questions that the project will need to address (see 
section on exploratory research). 

(j) Finalise (or make recdmmendatidns) about the project research design 
including data sources, data collection and analysis methods, critical project 
decision points, timetabling^ staffing^ resources and budget. 

1,3^2 Exploratory Cqnalltative) research methods 

Uses 

To ascertain the broad thrust of a course dffering in terms of scope, 
weights of emphasis to be given to segments of the program, depth of 
coverage within subjects and the more impdrtant aspects of subject 
sequencing. Prdperly handled, exploratory research is particularly useful when 
little information is available about an occupation and for obtaining an 
understanding of the structural complexities of an occupation as a prelude to 
identifying specific data needs for curriculum design. 

Data cbllectidn method 

The first step is to decide the research quisstidns which are to be explored 
in interviews with key informants or in group discussion. Research questions 
are the broad questions which guide and focUs the data collection and analysis 
phases. Examples are: 'In what ways is technological change altering the 
pattern of work skills in an occupation?' and 'What should be the central 
outcomes of the level of course being designed/reviewed?' The research 
questions should span the occupational/social context level (the wider forces 
which shape immediate knowledge needs or future needs which the educational 
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program would anticipate) arid the individual level, so as to allow student lieeds 
to be related to an understanding of how work iri the occupation is r rgahised; 

These research quisstidris are usually divided into finer questions such as 
those used as probes in interviews; Since this aim of exploratory research is "to 
explore, new questions are added as the process unfolds. the aim is to 
achieve a broad understanding of the occupation in terms of patterns or 
common themes but not to be deceived by initial appearances: to look for 
cracks or flaws in the emerging data and to follow up the questions that these 
pose. 

Sample 

This depends dri the scope of the project and the research questions being 
studied. The composition of the Sample should be decided according to what 
the researchers rieed to know. For example, if the problem for curriculum 
design is the existence of several generations of technology (latest to the 
oldest) in use in an occupation, sampling cases must include instances of all 
generations of technology so that curriculum developers can know the breadth 
of education required. 



The strength of exploratory research lies in its focus on the broader picture 
and on relating the educational needs at the individual level, to ari 
understanding of occupational structure and change. The method's sensitivity to 
detecting change and the educational implications of change is of great 
advantage in preventing the problem of preparing a course which is obsolete by 
the time it is implemented. 



imium on 



The method requires flexibility in implementation and places a pfer 
the ability to synthesise (join data) and to extrapolate (look beyond data). 
Since the usual data collection is by interview, skill is required in conducting 
relatively open and free-ranging interviews and analysing responses for main 
themes while remaining alert for cracks or flaws in the emerging data which 
should be followed up as leads for further exploration. 



Cost and duration 
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This is difficult to quantify since explbi-atbry research should sample enough 
'typical applications' of the work in the occupation, to provide an bveraii 
picture of how the occupation is organised (sti-uctured? and how it Is changing; 
Discidihg the minimum but adequate number of sample cases depends on the 
complexity of the occupation. Experience in TAFE applications on small, 
relatively homogeneous occupations suggests that the number of sample cases 
for focused interview and observation of work functions can be set at a 
minimum of 4-8. Once the broad picture has been obtained, the time required 
for curriculum development depends on the level of knowledge held by the 
syllabus writers. 

1.3.3 The^J 

Uses 

To obtain consensus from a group of 'experts' about the likely pattern of 
future events and to show where these experts disagree. 

Data collection method 

A series of questionnaires are sent to the 'expert' panel. Their responses 
are summarised and returned to the panel until patterns of agreement and 
disagreement become clear. Usually two or three rounds of questionnaire 
administration and analysis are sufficient. The method can also be used during 
a conference, e.g. where a questionnaire is written during the conference, 
administered, analysed and the pooled results returned to the group for rating 
or comment. 

Sample 

People who are in a position to comment on likely future trends. Thirty or 
more respdhderits. 

Advantages claimed 

The method accesses people without requiring them to meet iri persbh, 
saving travel time and cost. It describes what key decision makers believe is 
technologically possible given particular constraints. It gives participants equal 
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bppdrturiitiejs to set down their perceptions of future events. It eliminates 
those psychdldgical barriers to cdmmuhicatibn that can affect face-tb-face 
grbup interaction. Participants have time for thinking and reflection before 
giving final judgments. 



The method assumes that the achievement of consensus within the panei, 
renders the data more believable and that the arioriymbus responses are more 
likely to be more objective than those generated through face-to-face group 
prdcesses. The method may under- or over-estimate the likelihood ot future 
developments within ah industry because individual desires may become imposed 
on che forecasting process. Cross impact between specific forecasts can be 
difficult to assess. 

Applications 

The method is a promising means for assessing the directions which experts 
(or informed observers) believe an industry or social process (e.g. education) 
will take. Such directions could include what technological innovations from 
the range available are likely to be taken up, and in what time frame. Such 
information cannot* bf cburse, be proven. However, the ability of Delphi to 
detect patterns of change likely in the short term (1 to 5 years) could reduce 
the problem bf curriculum obsolescence, Delphi can also be used to scan che 
'here and now' and identify consensus on what are the matters of concern and 
the most promising solutions. 

Due to the dangers in forecasting* it would appear sound practice to join 
Delphi to a method which addresses the 'here and now' of work performed in 
an occupation. Because it accesses acknowledged experts, Delphi would work 
best in curriculum development for areas undergoing rapid change* when no one 
knows the shape of things to come. It would also be a useful component of a 
multiple method approach to rethinking a problem, e.g. the nature of trade 
training in the late 1980s, or examining approaches to distance education. 



Disadvantages 



i.3.4 The 




Uses 



To systematically develop a training program incorporating all aspects of the 
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curriculum (syllabus; assessment and teaching resources) based upon the 
cdmpiiation of a detailed task analysis (identification of job competencies) from 
the view of 'experts' via a structured g*^oup process. 

Data cdllectidil method 

Data on job competencies (duties, tasks, task elements) are collected from a 
panel of selectisd 'experts* in the occupational area being studied. In a 
structured workshop session managed by a skillful facilii:atorj the panel of 
experts is asked to define the occupational area in increasingly finer detail. 
Job categories are identified, followed by the major duties* cdmporient tasks 
and finally the skills, knowledge and attitudes related to the component tasks. 

The data are provided by the experts and recorded by the facilitator on 
cards or butcher's paper until all job competencies have been analysed and are 
presented in a DACUM chart which becomes the basis for curriculum 
development. 

Sample 

The DACUM workshop traditionally operates with about 6-14 experts, i.e. 
people who are actually performing or supervising the job under study. 
Participants in the workshop should be representative of different skill strata 
within the occupational group. 

Advantages claimed 

The method appears to be quick and easy to manage. It is fairly structured 
and business-like and seems to have appeal to industry experts. It focuses on 
what the practitioner needs to be able to do on the job. The method is 
systematic and embraces the training model used extensively by the US Armed 
Forces. As such it readily suits the established philosophies of many curriculum 
researchers in TAFE in Australia and provides ordered data which are conducive 
to full scale curriculum development activities where an entire instructional 
system is required. 

Disadvantages 

it has been argued that an instructional systems approach to training is too 
narrow and too inflexible to research effectively the learning requirements of 
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adult students, because it focuses bri an analysis of the job, rather than the 
learning styles of students; DAeUM faiis into this category. 

Since the classical form of DACUM draws data solely from industry experts, 
it has been argued that a DACUM derivisd curriculum may 'je poorly 
implemented by teachers, that is to say, because teachers and students are 
not a part of the analysis arid design process, they are not as committed to 
the curriculum when it is being implemented. To try to overcome this 
weakness, some TAFE versions of DACUM include teachers in the workshop 
process. Another weakness is that DACUM does not identify how an occupation 
is changing, though it is possible to add a concluding phase examining 'future 
trends'. 

Applications 

The DACUM method appears to be particularly suitable to researching 
occupational needs in established vocational areas, i.e. for reviewing or 
revisinj existing TAFE programs. This is the case because the DACUM process 
begins with the identification of job classifications and from there 
systematically analyses the tasks involved in performance in the job. 

It is further argued by some commentators that DACUM works best when 
applied to an occupational area which is comprised of a fairly homogeneous 
workforce. It would appear of:her more search-oriented methods could be more 
effective in exploring an occupational area or community need where no 
educational provision already exists. 

1.3.5^he^earch Coriference Method 
Uses 

The main uses of this method to date have been in organisatioi^al 
development, he. setting directions for improving organisational responsiveness 
to clients. In an educational context, the procedure, in essence, aims to assist 
oeople to make effective decisions now by clarifying what educational program 
they want to deliver in the future, taking account of likely future trends, and 
current constraints. 
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Data collection method 

The Search Gonference method is a flexible program containing a number of 
optional stages foiibwirig the first step which is a 'FUTURE SCAN' of forces 
which are shaping the future environment. The 'FUTURE SGAN' which focuses 
on events, trends or forces which are perceived to be shaping the future^ may 
iiiclude segments in which the conference considers the 'MOST DESlRABtE' and 
•MOST PROBABLE' futures. The central principle underlying the Search 
Conference method is that plans m.ade in response to a turbulent environment 
grow out of 

(ij shared ideals Or values about what are jhe important goal^ 
of planning for the future of the system in question 

(iij consideration of the constraints which must be taken into 
account (n realistic planning. 

Next, the workshop breaks into Small groups which take the product of the 
'FUTURE SCAN' and the 'MOST DESIRABLE' and the 'MOST PROBABLE' 
future. Or, depending on the Situation, the whole group may select 6-10 
changes they judge as important and spell out their likely effects taking into 
account 'CONSTRAINTS' and what to do about them. In some cases, the 
conference may examine the effects of changes or trends occurring inside the 
school or college ('INTERNAL SCAN') which are shaping its future. Another 
optional step, recommended in curriculum research applications, is the 
•HISTORieAL RECONSTRUCTION' - a rewinding of the clock to 'see where we 
have come from?', in order to see what is good arid should be preserved. The 
next step, consideration of present 'STRENGTHS AND WEAKNESSES' is to take 
stock of 'what we are good at* and 'what we are poor at?' By this time most 
of the key issues and options will have been identified and three Steps remain: 

(a) ISSUES AND PRIORITIES: Placing priorities on the issues 

identified during the conference. 

(b) ACTION PLANNING: Selection of Issues and the development 

of action plans by small groups. 

(c) FINAL SESSION: Report-back sessions on the action plans 

which have been formulated and discussion of 
implementatiori procedures. 

It is usual for the action groups to continue their work after the conference 
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ends and these groups may meet again 5r Feport to each other through a 
co-drdihatbr. 

Sample 

The essential requirement is representation of major 'stakeholders' (interested 
parties); In curriculum research applications these would include 
representatives of the occupational grouping, teachers, students, curriculum 
developers, administrators and other key decision makers in relation to 
curriculum implementation. Sample size around 36 is ideal but Search 
Conferences can be run with as many as i50 people. 

Advantages claimed 

The method provides 'future-oriented' data from which to design a 
curriculum. This is particularly important in design of vocational curricula for 
occupational areas undergoing change. The process tends to 'open the horizons' 
of participants {'FUTURE SCAN' leading to 'DESIRABLE FUTURES'), while 
keeping a firm footing in relation to implementation realities ('C0NSTRAINTS'). 
Because of the representation of 'stakeholders' in the conference and their 
involvement in the research and action planning stages, the method is said to 
offer enhanced prospects of implementation of outcomes. 

Sisadvantages 

As yet we do not know how effective the method is in analysing the detail 
of competencies required to perform work in ah occupational grouping. 

To effectively apply the method the facilitator requires a good understanding 
of how to work With a group in an exploration of the wider contextual forces 
impinging upon an occupational grouping. Moreover, each Search Conference 
must be tailored to the particular situation. This requires a detailed 
understanding of the method. 

Applications 

The ideal application is On an occupational grouping undergoing change. The 
method appears particularly suited to answering the broad questions in 
curriculum design such as course philosophy/rationale, aims, scope and course 
structure and can. it is said, be applied to trade and technician courses, as 
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well as short courses and organisational and personai development projects. 
The method is useful in circumstances where there are diverse views about 
what changes, if any, should take place in a curriculum. 

L3>6 Nominal Group Technique (NG T) 

Uses 

i To define a probjem 

. To gen^ate solutions 

. To establish priorities for action. 

Data collection method 

The Nominal Group Technique (NOT) requires a small group who meet in a 
'brainstorming' session. The process begins with individuals silently writing 
ideas. These are then listed, one from each member in turn, on large sheets 
of paper. The ideas are 'collapsed' to remove overlaps, if needed. The list of 
items is ranked to establish group cdhsehsus as to priorities. Any additions are 
made, then the list is subjected to a final vote. The process is structured to 
limit group discussion in the idea-generating phase (silent writing of ideas) and 
the listing of these ideas (by limiting discussion to clarification only). 

Sample 

NGT appears to work best with groups of 8-10 people (several groups can be 
run concurrently). Participants can be those in a position to possess expert or 
up-to-date knowledge of the area and/or those who will be required to 
implement the (group) decision. 

Advantages claimed 

The method appears to be especially effective when the participants are 
strongly motivated to attempt the task (define the problem, or solve a known 
problem) and can tolerate the rules of the method, e.g. the rule of 'no 
discussion except for clarification'. The method claims to encourage equal 
participation and reduce the effects of one or two participants dominating the 
session. Moreover, because the method uses different procedures in the 
idea generation and idea evaluation phases, it is claimed that the method 
improves both the speed of generation of ideas and the quality of their 



evaluatiorii The method also gives a numericai weighting to the output and so 
can show what facets of the problem are seen as central, or what solutions 
seem most promisihg. 

Disadvantagies 

A key to the method is the framing of the MOT question which focuses the 
group's effc5rts. The NGT question can be specific, e.g. 'What subjects in this 
trade course should be reviewed?' or broad, e.g. 'What will the plastics 
worker of the 1980s be required to know?' B-at, unless the NGT question is 
appropriate, the process can be impaired. Some experience is needed, it 
appears, in formulating an NGT question which is effective^ central to the 
issue being addressed and one that participants can relate to and understand. 
The level of abstraction (generality) involved in answering an NGT question 
may need to be specified. The method may be less suitable for occasions 
when sharing and evaluating information, in open discussion^ are important in 
both generating solutions and setting the groundwork for smooth 
implementation. The method can generate frustration if applied in pn 
insensitive, excessively rule-guided way to a group who really need to talk out 
a problem as a prelude to tackling it in a systematic manner, 

1.3.7 Force Field 
Uses 

To generate strategies for implementing change situations by identifying the 
forces, both negative and positive, that are retarding change. 



Data collection method 

For each problem area or concern, the forces which might help in s5lving 
the problem are listed opposite the forces which might hinder the 
problem-solution process. The 'helping' and 'hindering' forces are then ranked 
from most to least important. After goals have been decided, action can be 
taken to either strengthen 'helping' forces or to weaken 'hindering' forces, or 
both. 

Advantages/disadvantages 

To our knowledge the technique has not been used in curriculum 
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development in TAFE. The method, in analysing change situations, has the 
advantages of offering a systematic approach to prdblem solution by teasing 
but the helping/hindering forces and identifying the points at which change 
should be applied. It therefore provides the straitegy for change, e.g. 
strerigtheii specific helping forces, weaken specific hindering forces. 

While not a curriculum research method in the formal sense, it does appear 
to offer a useful technique, if used following an NGT which identifies the 
problem areas, to tease apart the heloing/hindering forces which bear on the 
identified problem. In this application, the method would have the advantage 
over a solution-generating form of NGT (as distinct from a problem-identifying 
form) of providing greater opportunities for the group to share and interact on 
perceptions and ideas, while retaining the 'hitch-hiking' (building on the Ideas of 
others) advantages of NGT. 



Sample 

Groups of 4-8 persons. 
Applications 

Force Field Analysis is mainly a tool for providing directions in situations of 
complexity and resistance to change, e.g. a planning group wishing to foster 
the uptake of an innovation (technical or educational) might use the method to 
analyse implementation difficulties and adopt a strategy for change. Similarly, 
a group wishing to introduce a needed change might use the method to show 
where effort (leverage) might be applied to best effect. 

1.3.8 Critical incident Technique 

Uses 

To obtain a description of behaviours (completed action sequences) that are 
'critical', in terms of either success or failure, to tho performance of a task or 
critical gaps in knowledge, i.e. 'never do X when Y is happening'. 

Data Collection method 



'Gritical incidents' data can be collected by observers (e.g. researchers, 
supervisors) or the job holder. Data can include: what led up to the incident; 



what actiahs were effective (or ineffective); perceived consequences of the 
action; and whether the consequences were within the control of the job 
holder. Collection procedures can include recall, observation, interviews o? 
questionnaires. 



Sample 
Job holders. 
Sample size 



This depends on the problem but sample size can be reduced through 
stratification with fewer cases treated in greater detail. 



Since the data are observable behaviour, the technique is said to maximise 
objectivity while reducing subjective components of inference and 
interpretation. That is, the method draws out factual information of the 'how 
to/how not to' type, rather than the 'I think type. 

Applications 

The technique is suitable for eliciting practical information about working 
with equipment, processes or people. It can be used to decide weights of 
emphasis in curriculum design and assessment decisions (what content is 
•critical'). It has wide application in studying social skills needed in technical 
arid further education. 

The concept of 'critical incidents' (i.e. do's and dont's) can be used in other 
research methods, e.g. an NGT focused on a part of the work of an 
occupation. In student questionnaires, questions can be asked on (1) what 
aspects of the course were critical and positive in helping the student to 
perform key task sequences successfully, or (2) what parts of the course were 
irrelevant, or (3) what task were you required to do but were unprepared for"^ 
This information would, of course, need to be set into perspective by analysis 
of the student's background such as type of work performed by employer 
Critical incidents collected across a pool of respondents could be the basis of 
an industry or student questionnaire. 
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Ehiratibn and cost 

As yet, there are no case study applications in TAFE of this method. The 
method c^.n be time consuming in data collection and analysis but appears 
worthwhile in terms of the practical relevance of the data to successful work 
performance. 
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2. SUMMARY OF THE FACILITATION OF CURRICULUM RESEARCH 
WORKSHOPS IN TAFg 

2.1 ABSTRACT 

The processes entailed in facilitating or leading workshops or seminars for 
technical and further education curriculum research were examined by a 
modified Deiphi process (questionnaire with feedback of results to participants) 
followed by a two-day workshop. Participaints in the study were all 
experienced curriculum development specialists with knowledge of one or more 
of the following group process methods: Search Conference, Nominal Group 
Technique, DACUM (developing a curriculum) and derivatives of DACUM, the 
Critical Incident Technique and other group discussion techniques. The study is 
a companion to the examination of curriculum research methods by the same 
authors titled: "TAFE Curriculum research: a review of group process methods'. 

The facilitator role is analysed in terms of the value orientations 
(preferences) and social rules (guides for action) used by experienced TAFE 
facilitators/group leaders. 

The results show that the facilitation/group leadership process requires 
complex interpersonal skills including the ability to set clear directions in a 
hon-authoritarian manner, the ability tc negotiate flexibly with group members, 
to face and resolve conflict and to .tldtivate groups. The social rules and 
value orientations adopted by the facilitators studied are presented in detaih 
A model is presented of the general stages and steps through which curriculum 
research workshops progress. The model describes the value orientations and 
social rules which appear appropriate to each stage. A guide to the selection 
of group process methods for curriculum research in vocational education is 
included. 

2.2 INTRODUCTION 

Gathering qualitative data Using curriculum research group process methods 
places the facilitator (or group leader) in a crucial position in relation to 
quality of the workshop output. It is therefore important to understand the 
facilitator role in the implementation of curriculum research group process 
methods. This section is a summary of a study of the facilitation of 
curriculum research workshops (Anderson and Jones, 1986) which examined how 
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a group of experienced TAFE curriculum developers planned and conducted 
group process methods for TAFE applications. These approaches were analysed 
in terms of the value orientations (preferences for things and actions) and 
social rules (guides for social behaviour) found to be effective in conducting 
successful curriculum research wbrkshops in TAFE. The intending workshop user 
may wish to refer to the above-mentioned study for more detail about the 
complex skills required to conduct group process methods and for an 
explanation of the concepts of values arid social rules. The main findings from 
that study are summarised below. 

2.3 GENEI^At STAGES IN eURRieUtUM RESEARCH WdRKSHOPS, AND 
VALUE ORIENTATIONS AND SOCIAL RULES APPROPRIATE TO EACH STAGE 

Table 2 gives a general picture of a curriculum research workshop in terms 
of the main stages arid steps^ and the values and rules appropriate to each 
stage. It is not meant to relate to any particular curriculum research group 
process method but to highlight the general principles which should be 
addressed when desigriirig and implementing curriculum research group process 
methods. The data on which the table were based were derived using a 
modified Delphi process (eliriting value orientations and social rules used in the 
conduct of curriculum workshops)^ followed by a two-day seminar exploring the 
complexities of the facilitator roie. This method of leading irito a 
workshdp/ssmiriar from a Delphi questionnaire had two advantages. First, it 
was an excellent 'warm-up' procedure for getting participants to think about 
the issues. Second, it gave each participant a look at what the other 
respondents saw as the important dimerisioris of the facilitator role. 

An important point to bear in mind about the concept of social rules is that 
rules can be made, broken and renegotiated (Harre and Secbrd, 1972). Rules for 
making and breaking the rules of curriculum research and development 
workshops depend on the curriculum research methods being used and on the 
situation in which they are being applied, i.e. on the dynamics of the seminar 
itself. The philosophies of curriculum research methods differ but, in 
implementation, they share as common ground the skills of working with people 
in groups. The purpose of Table 2, therefore, is to show the common themes 
for working successfuily with people in TAFE curriculum research workshops. 



TABLE 2 

STAGES AND STEPS IN CURRICULUM RESEARCH WORKSHOP 
PLANNING AND IMPLEMENTATION AND FACILITATORS' VALUE 
ORIENTATIONS AND SOCIAL RULES (SUMMARY) 



Stages and steps Value orientations/action guides 



Stage i: Planning 

a) Research the issues 



b) (let aims clear 



c) Select or design 
workshop 



d) Choose participants 
carefully 



Know this background. Extract the 
the issues. Know the politics 
(the 'hidden agendas') affecting 
the project brief or likely to 
generate tension within the 
workshop. 

Understand the issues and problems 
which will affect implementation 
of the outcomes. 

Know what the sponsor wants. 
Know what you are going to do in 
the workshop. Be able to 
communicate this clearly. 

(See Table 1 for questions 
which guide selection of 
various methods.) 

Know the method thoroughly. 
Be able to explain the method 
clearly and succinctly, 
Lricluding its 'ground rules'. 
Be prepared to be flexible 
in implementation. Know where 
you can bend the rules. Have 
contingency options. 

Get the 'right' people for the job, 
i.e. those in a position to 
know the area being studied. 
Choose participants with a 
good cross-section of 
knowledge/skills/awareness. 



Stages and steps 



Value orientations/action guides 



e) Check out, set up 
venue 

THE WORKSHOP 



Aim for comfort, ease of 
interaction. Allow for 
socialising* 



Stage 2: Orienting participants to the task and how it will be 
achieved (the method or process) 

a) Welcome participants 



Be friendlyi Put people at ease. 
Begin building 'team spirit' and 
group cohesion. 



b) Explain aims/task/ 
process/outcomes 



c) Gheck out participants' 
expectations 



dj Participants acquire 
skills and knowledge 
about the process 



Brief and orient participants 

to the planned process. Be 

clear and brief in explanations. 



Help people to 'see' where 
they are going. 

Listen carefujly (throughout the 
workshop). Show that you are 
listening. Be open^ State your 
expectations. Indicate what is 
ab[e to be changed and what is 
D9^- J^??P9G*^ sensitively 
and perceptively to group and 
individual needs. Defuse tensions 
early. Set up a supportive 
atmosphere. Value all 
contributions equally. Do not 
be authoritarian. Avoid 
'power-tripping'. Attend to any 
problems which participants 
have in regard to the process 
being used. Be prepared to shift 
tack (be flexibly responsive to 
group and individual needs). Use 
consensus to make decisions. 

Participants need to understand 
the process in which they are 
participating. Teach the required 
skills to this participants, 
if necessary. 
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Stages and steps 



Value orientations/action guides 



Stage 3; Build momentum 

Attend to any problems which 
participants have in relation to 
the process being used. 
Let the group set the pace. 
Adjust the rules to the group. 
Make decisions by consensus. Do 
not value the task over people's 
needs. Maintain group cohesion. 
Lead by enthusiasm. Create a 
feeling of energy. Encourage 
participation. Be skilled^in 
getting information from the 
group. Don't be intrusive 
but draw out silent members. 
Handle confrontations when they 
arise, sensitively and 
tactfully. Avoid emotional 
roadblocks to communication. 

Stage 4: The workshop in 'full-swihg' 

Be prepared to step aside if 
the group is working well, i.e 
has 'niatured* to the task. 
Be prepared to change direction 
if the group becomes 
bogged down. Assert control 
where appropriate. 

Attend to 'people needs'. 
Watch participants' comfort 
needs. Monitor time use. 
Schedule breaks when 
appropriate. Maintain group 
spirit and cohesion. 

In jhe task work, the 
faciiitator may be more 
demanding (not satisfied with 
partial solutions). Monitor 
time usage. 

After the group leaders are 
selected (preferably by the 
group), ensure that they and 
the group understand the task. 



a) Plenary (whole group) 
sessions. 



b) Small group sessions 
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Stages and steps 



Value orientations/action guides 



Stage 5: Goncluding the workshop 



a) Summarise 



Summarise the workshop 
findings. 



b) Achieve closure 



Leave people with a sense of 



accomplishment. 



c) Get commitment 



Get commitment to outcomes 
(especially important with 
the Search Conference method). 



Stage 6: Follow-up 



Document and distribute reports 
of the workshop to participants. 



2.4 Common themes 

These themes or value orientations are as follows: 
A humanitarian focus 

Although facilitators vary in the way they approach the task work of the 
seminar or w^brkshop, they appear to be very person centered in their approach 
and attend closely to the personal needs of the workshop participants. It 
appears that when personal needs are fulfilled the task work flows with greater 
smoothness. 

A task-oriented focus 

A good facilitator achieves the task through making the aims, expectations 
and outcomes of the workshop clear, attending to individual and group needs 
sensitively, defusing and/or resolving conflict, watching the use of time and 
using control where appropriate. 

An egalitarian focus and avoidance of *power-tfipping* 

A good facilitator avoids ^power-tripping'. This is important because the 
facilitator possesses a degree of power which, unless handled sensitively, can 
have damaging effects. This power can be expressed in an autocratic 



overbearing manner used in an attempt to imprisss the group, oi by adoption of 
a judgmental manner, or by showing insensitivity to the needs of participants 
and to what they are saying, or by driving the group too hard. These 
manifestations of *power-tripping' create distance between the facilitator and 
the group. In attempting to avoid the negative expression of the power 
inherent in the role, facilitators adopt a humanist, caring approach, take care 
to negotiate the program and any changes with the group, and create the 
conditions that allow the group's energies to be relersed arid the guidance to 
allow them to work effectively ori the task. The egalitarian value adopted by 
facilitators is revealed in the rules of treating all participants as equals, 
valuing contributions equally, transmitting empathy, and making decisions by 



consensus. 

A leadership focus 



Facilitators appear to adopt the conception of a leader as one who sets 
clear directions, is flexibly responsive to group and individual needs, and is 
prepared to step aside when the workshop momentum is progressing well. 

The ability to motivate groups 

Facilitation requires an ability to generate, unlock, or inspire energy and 
interest within the group. The mearis for achieving this are by careful design 
of the workshop and selection of participants, the adoption of the values and 
rules embodied in the first two themes mentioned above, together with the 
transmission of enthusiasm about the worth of the project, transmission of 
positive expectations about the outcome, the display of faith in the group (e.g. 
through valuing group contributions), and a preparedness to adapt the process to 
the needs and interests of the group (by being sensitive and perceptive of 
group and individual needs). 

Good faeiiitation requires excellent searching and communication skill 

Searching skills include the ability to tease out the issues beforehand 
including the politics and 'hidden agendas', the implementation realities and 
problems which bear upon curriculum development, and the ability to plan the 
workshop accordingly. Communication skills include the ability to listen 
carefully, particularly to the 'music behind the words', and to be clear and 
succinct. 
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These themes are taken ujD in Table 2 which shows the general stages and 
steps in the evolution of curriculum research workshops^ and the values and 
social rules which appear appropriate to each stage. The skills listed below 
are important in striving for the ideal of a democratic leadership style. 

Skills required by facilitators/group leaders 

This section summarises the skills required by the facilitator. 

Communication skills 

The facilitator needs a high level of skill in communicating effectively. 
This requires, among other things, the ability to have an open mind and to 
listen carefully and actively. One way of doing this is to show that you are 
trying to follow what people are saying and that you value whet they are 
saying. Maintain eye contact with the person who is speaking. Be sensitive to 
the verbal and non-verbal cues from the group. 

Make the relevance of activities clear (in relation to the overall task). 
Avoid jargon and be precise. Have an open mind. 

Ete tolerant, respectful and appreciative 

Acknowledge the views expressed by participants even though they may not 
be personally acceptable. Be democratic. Value people for themselves. Avoid 
put-downs, indicate that you respect participants' expertise. Be prepared to 
invest time in dealing with emotive issues so that they do not waste time 
later. Allow anyone to intervene at any time arid acknowledge their concern, 
especially when you do not immediately act on it. Indicate your willingness to 
be interrupted. 

Be casual, relaxed^ open arid sincere 

Be casual, relaxed, low-key arid concerned to put group members at ease. 
Be open to others. Answer questions frankly. Say what you are ddirig and why 
you are doing it. Be seeri as fair minded, i.e. to seek other views without 
passirig judgment. Keep formality to a minimum. Be honest and sincere in 
interactions with group members (do not work behind a facade). 
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Transmit empathy 



Try to see things from the other's perspective: to stand in the other's 
shoes. Try to blend in with the group (including style of dress). Do not be 
seen as aloof. 

Be friendly and supportive 

Be friendly, supportive^ encouraging and warmly responsive to people. Be 
attentive. 

Adopt a quiet, confident and non-defensive manner 

Be quiet (i.e. not dominant or authoritarian), and project a confident manner 
and a sense of direction. 

Be fair 

Show appreciation of contributions. Be seen as fair minded. 
Be flexible 

Adjust the workshop process to the aims of the meeting and to the needs 
and pace of the group. Pick up important issues raised by the group. Allow 
regular tirrle-dut from the task work to examine the process (by which the task 
is being achieved) and correct any problems. Value the group's efforts in this 
regard. Since there may be conflict about the process itself, allow questioning 
of the process. Bend the rules or change the direction or style of operation 
according to the situation. 

Be aware of the importance of the group's feelings and pick up group issues 
and suggestions 

Feelings are an important source of information. Be sensitive to the mood 
or feeling within the group. Have the ability to handle feelings and 
confrontation well (e.g. by defusing potentially tense confrontations between 
participants which is important in overcoming emotional road-blocks to 
communication). Conflict can be an important indicator of an underlying 
problem, which may relate to how a task should be approached and if 
addressed tactfully, can allow the group to become productive. 
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Work toward shared understanding 

Work toward a clear, shared understanding with the group. 
Show faith in the group 

Trust people to take responsibility for their own lisarnirig. Communicate, 
verbally and ndh-verbally, positive messages about the outcome. Try to inspire 
interest arid create a feeling of energy. 

Use control when appropriate 

Deal with dissent or resistance immediately it surfaces - listen especially 
carefully as there may be a misunderstanding which you can clear up or you 
may be doing something which needs to be put right immediately. Notice 
when tensions are starting to arise and defuse them, e.g. using humour to 
lighten the situation or by introducing a break in proceedings. Just 
acknowledging the tensions may sometimes be sufficient. 

Counter productive ^pects^^f Vacillation 

Several destructive aspects of facilitation are summarised below. 
Po we r-t rippi n g 

Avoid being overbearing, dominating, too directive or autocratic. Do not sit 
in judgement on the views expressed, or indicate that alternative ways of going 
about the task are a poor choice or override individual participant's needs. Do 
not try to impress the group with knowledge. Do not push tiie group to your 
solutions or push the group too hard. 

Failure to provide clear direction 

Do not give confusing instructions, set Unrealistic goals, allow aimless 
discussion or let the process continue unmonitored. 

Failure to motivate groups 

Do not allow participants to become restless or dissatisfied, allow reporting 
to go on and on, make comments that are negative or critical, engender group 
hostility or resistance to learning, or break the flow of the group. 
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Lack of sensitivity, insight, care and consideration of peSple and their 
feelings 

Do riot over-emphasise the task over people's needs; Don't leave people 
wondering what is wrong with themselves. Bo not cut people down. 

Coi^clusioh 

The foregoing shows that considerable skill is needed to effectively faeiHtate 
or lead a curriculum research workshop. This is especially the case when 
conflict situations arise, hence planning is important:. 

There is a need to learn more about how conflict can be handled not by 
choking it off or sweeping it away but recognising it, working with it, and 
trying to find solutions which relate to the goal of achieving quality and 
relevance in the curriculum. The facilitator must be able to recognise 
irreconcilable conflict for what it is and abandon or postpone further work 
until the conflict is resolved. 



3. SUMMARY OF 'TAFE CURRICULUM RESEARCH: A REVIEW 0F CROUP 
PROCESS METHODS. RESEARCH DESIGN' 

3.1 ABSTRACT 

This report set. out the research method and evaluation criteria used in the 
TAFE National Centre for Research and Development study of occupational 
research methods, conducted during 1983 and 1984. The methods studied were 
those seen to hold promise of speeding up the data collection and analysis 
phase of vocational curriculum research. This paper describes the concept of a 
•resource bank of methodologies' which includes description of method, human 
and material resources required to use the method effectively, the suitability 
of the method to particular TAFE study areas, how to use the method, personal 
and professional skills required to use the method and constraints and pitfalls 
to be considered in using the method. 

The criteria for selection and evaluation of case studies and their 
documentation is described. The evaluation criteria cover process criteria such 
as: duration, cost, expertise and manpower required, acceptability, capacity for 
blending with other methods and hardware requirements; and outcome criteria 
such as: relevance and utility of the method's results, quality of data collected 
capacity of the method to forecast and to induce change, and transferability of 
results. 

3-2 SUMMARY OF EVALUATION CRITERIA 

The criteria that have been established for determining the worth of 
methodologies used in the various case studies are set out below in two 
classes: process criteria and outcome criteria. Process criteria concern the 
resources required (both human and material) to effectively use the method 
and the processes adopted. Outcome criteria concern the nature, range, quality 
and utility of curriculum products developed as a result of the use of the 
method. 

a) Process Criteria: 

I. Duration of the-^gmthori 

* units"of°tiie"^^'^^'^ '° "'^^^"^^d in 
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. time lapse from start to finished product, measured in 
Units of time. 

2. Cost of using the method, measured in $. 

3. Nature and level of expertise required to effectively use the method - 
expertise in research, curriculum development^ teaching practice, interpersbrial 
skills, and industry knowledge may all be appropriate. Separate measures of 
ievel of expertise for each of the factors listed above may need to be derived 
- these measures would be relative, and would be derived largely from 
subjective data provided by selected 'experts' in relation to each factor. 

4. Number of personnel required to use the method, measured in number of 
fulj-time equivalent people ordinarily engaged in conducting the method. 

5. Acceptability of the method to the TAFE environment - this criterion 
included such factors as: (a) degree of involvement of teachers and other 
stakeholders in the process, (b) degree of perceived departure of the method 
from the 'normal' methods Used and 'known' in TAFE, and (e) suitability of the 
method to the extant political and economic environment in TAFE. 

This criterion is necessarily descriptive and would, perhaps, best be conveyed 
by identifying the 'bottom-line' of each factor - e.g. it may be judged that a 
method is not acceptable unless it involves all the teachers of a certain rank 
and status in the data collection and/or analysis, process. 

6. Capacity of the method to fit into a modular curriculum research model 
- that is to say, one where the method is readily partitioned in phases or parts 
in order that its part may be selected and used in concert with parts of 
another method in combinations appropriate to the circumstances at hand. 

7. Hardware requirements of method, in terms of data and computer 
facilities, accommodation and travel requirements. 

b) Outcome criteria 

1. Relevance of data collected arid analysed using the method. This couid 
be measured by identifying actual purposes to which data are pUt in the 
curriculum development process - relevance could be measured by a simple 
3-point nominal scale, such as (ij relevant and necessary, (2) relevant and 
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sufficient, and (3) irrelevant. 



2. Utility of data collected and arialysed as a basis for curriculum 
development including curriculum delivery and implementation. This could be 
measured by a panel of expert curriculum developers making subjective 
judgements based on their experiences. The panel would assess the utility of 
data of different kinds, and which had been analysed in different ways, for the 
purposes of curriculum development, delivery arid implementation. 

3. Capacity of the method to forecast future needs. Due to the difficulty 
in determining a reliable means of forecasting and hence the difficulty in 
developing a measure for the success of such forecasting, the simple cuestion 
that can be asked here is: 'Does the method provide any data at all about 
future rieeds?' 

4. Capacity of the method to bring about the extent of change that is 
iriterided by the change agents (researchers, curriculum developers, 
administrators, etc). This criteribri is descriptive and would depend upon the 
perceptions of 'experts' in the change process. 

5. Transferability of the outcomes or presentation of findings of the method 
arid its analysis. A method is necessarily more generally Useful to all TAFE 
Authorities if the data which it presents and analyses are of a format which is 
cbrisisterit with the educational and administrative conventions and requirements 
of each TAFE Authority. 

The evaluation dimension for this project would result in the selection of a 
number of case studies which could form the focus of the developmental 
dimension of the study. The case studies were to be selected by the research 
team, after cdrisultatibri with the Project Advisory Committee on the basis of 
the evaluation criteria outlined above, and intuitively with a regard for the 
potential pay-bff of the case study and the 'convenience' in negotiating the 
case study arrangements. 
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